2025-11-13 08:53:01,000 [ 295421 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-11-13 08:53:01,000 [ 295421 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:79, check_args_and_update_paths) 2025-11-13 08:53:01,000 [ 295421 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:90, check_args_and_update_paths) 2025-11-13 08:53:01,000 [ 295421 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:92, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_ipjoaj --privileged --dns-search='.' --memory=30709026816 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel1_1.jsonl --report-log-exclude-logs-on-passed-tests test_keeper_memory_soft_limit/test.py::test_soft_limit_create test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache Test order randomisation NOT enabled. Enable with --random-order or --random-order-bucket= rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: timeout-2.3.1, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0, random-order-1.1.1 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [3 items] scheduling tests via LoadFileScheduling test_reload_clusters_config/test.py::test_delete_cluster test_keeper_memory_soft_limit/test.py::test_soft_limit_create [gw3] [ 33%] SKIPPED test_keeper_memory_soft_limit/test.py::test_soft_limit_create [gw0] [ 66%] PASSED test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster [gw0] [100%] FAILED test_reload_clusters_config/test.py::test_update_one_cluster =================================== FAILURES =================================== ___________________________ test_update_one_cluster ____________________________ [gw0] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_update_one_cluster(started_cluster): send_repeated_query("distributed") send_repeated_query("distributed2") > assert get_errors_count("test_cluster") > 0 E AssertionError: assert 0 > 0 E + where 0 = get_errors_count('test_cluster') test_reload_clusters_config/test.py:202: AssertionError ------------------------------ Captured log call ------------------------------- 2025-11-13 08:55:58.990000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:56:11.243000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:56:23.496000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:56:35.749000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:56:48.051000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:57:02.060000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:57:14.312000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:57:26.564000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:57:39.317000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:57:51.570000 [ 633 ] DEBUG : Executing query SELECT count() FROM distributed2 SETTINGS receive_timeout=1, handshake_timeout_ms=1 on node (cluster.py:3731, query_and_get_error) 2025-11-13 08:58:03.820000 [ 633 ] DEBUG : Executing query SELECT errors_count FROM system.clusters WHERE cluster='test_cluster' and host_name='node_1' on node (cluster.py:3648, query) ---------------------------- Captured log teardown ----------------------------- 2025-11-13 08:58:04.099000 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/.env --project-name roottestreloadclustersconfig-gw0 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml stop --timeout 20] (cluster.py:121, run_and_check) 2025-11-13 08:58:11.496000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:11.497000 [ 633 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:121, run_and_check) 2025-11-13 08:58:11.508000 [ 633 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/.env --project-name roottestreloadclustersconfig-gw0 --file /ClickHouse/tests/integration/test_reload_clusters_config/_instances-1-gw0/node/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml down --volumes] (cluster.py:121, run_and_check) 2025-11-13 08:58:12.011000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:12.011000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:12.011000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Removing (cluster.py:147, run_and_check) 2025-11-13 08:58:12.011000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-node-1 Removed (cluster.py:147, run_and_check) 2025-11-13 08:58:12.011000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:12.011000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopping (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Removing (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Removing (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Stopped (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Removing (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo1-1 Removed (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo2-1 Removed (cluster.py:147, run_and_check) 2025-11-13 08:58:12.012000 [ 633 ] DEBUG : Stderr: Container roottestreloadclustersconfig-gw0-zoo3-1 Removed (cluster.py:147, run_and_check) 2025-11-13 08:58:12.013000 [ 633 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw0_default Removing (cluster.py:147, run_and_check) 2025-11-13 08:58:12.013000 [ 633 ] DEBUG : Stderr: Network roottestreloadclustersconfig-gw0_default Removed (cluster.py:147, run_and_check) 2025-11-13 08:58:12.013000 [ 633 ] DEBUG : Cleanup called (cluster.py:851, cleanup) 2025-11-13 08:58:12.031000 [ 633 ] DEBUG : Docker networks for project roottestreloadclustersconfig-gw0 are NETWORK ID NAME DRIVER SCOPE (cluster.py:830, print_all_docker_pieces) 2025-11-13 08:58:12.047000 [ 633 ] DEBUG : Docker containers for project roottestreloadclustersconfig-gw0 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:838, print_all_docker_pieces) 2025-11-13 08:58:12.062000 [ 633 ] DEBUG : Docker volumes for project roottestreloadclustersconfig-gw0 are DRIVER VOLUME NAME (cluster.py:846, print_all_docker_pieces) 2025-11-13 08:58:12.062000 [ 633 ] DEBUG : Command:[docker container list --all --filter name='^/roottestreloadclustersconfig-gw0-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:121, run_and_check) 2025-11-13 08:58:12.077000 [ 633 ] DEBUG : Unstopped containers: {} (cluster.py:865, cleanup) 2025-11-13 08:58:12.077000 [ 633 ] DEBUG : No running containers for project: roottestreloadclustersconfig-gw0 (cluster.py:879, cleanup) 2025-11-13 08:58:12.077000 [ 633 ] DEBUG : Trying to prune unused networks... (cluster.py:885, cleanup) 2025-11-13 08:58:12.092000 [ 633 ] DEBUG : Trying to prune unused images... (cluster.py:901, cleanup) 2025-11-13 08:58:12.092000 [ 633 ] DEBUG : Command:[docker image prune -f] (cluster.py:121, run_and_check) 2025-11-13 08:58:12.114000 [ 633 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:145, run_and_check) 2025-11-13 08:58:12.115000 [ 633 ] DEBUG : Images pruned (cluster.py:904, cleanup) 2025-11-13 08:58:12.115000 [ 633 ] DEBUG : Trying to prune unused volumes... (cluster.py:910, cleanup) 2025-11-13 08:58:12.115000 [ 633 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:121, run_and_check) 2025-11-13 08:58:12.130000 [ 633 ] DEBUG : Stdout:1 (cluster.py:145, run_and_check) 2025-11-13 08:58:12.130000 [ 633 ] DEBUG : Volumes pruned: 1 (cluster.py:915, cleanup) ----------------- generated report log file: parallel1_1.jsonl ----------------- ============================== slowest durations =============================== 136.99s call test_reload_clusters_config/test.py::test_delete_cluster 125.05s call test_reload_clusters_config/test.py::test_update_one_cluster 34.33s setup test_reload_clusters_config/test.py::test_delete_cluster 26.27s setup test_keeper_memory_soft_limit/test.py::test_soft_limit_create 8.03s teardown test_reload_clusters_config/test.py::test_update_one_cluster 1.57s teardown test_keeper_memory_soft_limit/test.py::test_soft_limit_create 0.22s call test_keeper_memory_soft_limit/test.py::test_soft_limit_create 0.00s teardown test_reload_clusters_config/test.py::test_delete_cluster 0.00s setup test_reload_clusters_config/test.py::test_update_one_cluster =========================== short test summary info ============================ FAILED test_reload_clusters_config/test.py::test_update_one_cluster - Asserti... PASSED test_reload_clusters_config/test.py::test_delete_cluster SKIPPED [1] test_keeper_memory_soft_limit/test.py:46: Disabled for sanitizers ============== 1 failed, 1 passed, 1 skipped in 306.49s (0:05:06) ============== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 492, in subprocess.check_call(cmd, shell=True, bufsize=0) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_ipjoaj --privileged --dns-search='.' --memory=30709026816 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=5ccda723c1fc -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=d862517635bf -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=1 --color=no --durations=0 --report-log=parallel1_1.jsonl --report-log-exclude-logs-on-passed-tests test_keeper_memory_soft_limit/test.py::test_soft_limit_create test_reload_clusters_config/test.py::test_delete_cluster test_reload_clusters_config/test.py::test_update_one_cluster -vvv " altinityinfra/integration-tests-runner:226bfaf75ac1 ' returned non-zero exit status 1.